During this class, we will need tensorflow 2, which is not installed in miniconda by default. You can install it using the anaconda navigator as follows:
Step 1: Open anaconda navigator.

Step 2: Click on the environments tab and search Tensorflow 2 package.

Step 3: Select Tensorflow 2 package and install it.

(Images are taken from this tutorial)
You can also install tensorflow and other Python packages using pip. For that, you need to open a terminal window, and assuming that your Python environment is active in that window, you can install tensorflow typing: pip install tensorflow.
The Moon dataset is an artificial dataset with two intertwined moon shapes belonging to two different classes.
The dataset is composed of 10000 data points with the following features:
So, our data matrix is of size (10000, 2); that is (nb_data_points, nb_features).
%matplotlib inline
import matplotlib.pyplot as plt
plt.rcParams['figure.figsize'] = [10, 10]
from sklearn import datasets
# Here we add some noise so that some points are a bit outside of the moon shapes
X, y = datasets.make_moons(n_samples=10000, noise=0.05)
print("The data shape is {}\n".format(X.shape))
print("The first 5 data points are \n{}\n".format(X[:5]))
print("The first 5 labels are {}\n".format(y[:5]))
print("The last 5 data points are \n{}\n".format(X[-5:]))
print("The last 5 labels are {}".format(y[-5:]))
The data shape is (10000, 2) The first 5 data points are [[ 0.90795979 0.33809101] [-0.99327662 0.32234129] [-0.96663396 0.19647839] [ 0.89584431 -0.45014748] [-0.97587863 0.08234114]] The first 5 labels are [0 0 0 1 0] The last 5 data points are [[ 0.64584247 0.81012225] [ 1.69086985 -0.12902535] [-0.23833011 1.02212343] [ 0.91640458 0.29492132] [ 0.90155487 0.51730699]] The last 5 labels are [0 1 0 0 0]
Now, let us visualise what this dataset looks like.
import seaborn as sns
import numpy as np
ax = sns.scatterplot(x=X[:,0], y=X[:,1], hue=y)
Now that we have our dataset, we will create a function to plot the decision boundary of our models and use it to compare the behaviour of a logistic regression and a deep learning model. Note that logistic regression is another name for our familiar delta rule when the sigmoid activation function is used.
Nothing to do here, but you can explore this function to see what it does if you are curious about it.
Note that this part of the class is inspired by scikit learn example on multinomial logistic regression.
# This function will plot the decision boundaries of any model given our 2D dataset. We will use this function many times in this class.
def plot_decision_boundaries(X, y, clf, step=None):
# create a mesh to plot in
h = 0.02 # step size in the mesh
x_min, x_max = X[:, 0].min() - 1, X[:, 0].max() + 1
y_min, y_max = X[:, 1].min() - 1, X[:, 1].max() + 1
xx, yy = np.meshgrid(np.arange(x_min, x_max, h), np.arange(y_min, y_max, h))
# Plot the decision boundary. For that, we will assign a color to each
# point in the mesh [x_min, x_max]x[y_min, y_max].
Z = clf.predict(np.c_[xx.ravel(), yy.ravel()])
# Put the result into a color plot
Z = Z.reshape(xx.shape)
plt.figure()
plt.contourf(xx, yy, Z, cmap=plt.cm.Pastel1)
plt.axis("tight")
# If we are plotting the decision boundaries sequentially, we mention the corresponding epoch
if step is not None:
plt.title("Decision boundaries at epoch {}".format(step))
# Plot also the training points
colors = ["blue", "orange", "green"]
for i, color in zip(range(3), colors):
idx = np.where(y == i)
plt.scatter(X[idx, 0], X[idx, 1], c=color, cmap=plt.cm.Pastel1, edgecolor="black", s=20)
plt.show()
Let us train a logistic regression model and plot its decision boundaries after training.
from sklearn.linear_model import LogisticRegression
# delta rule is used in the next line of code to fit the model to the data
clf = LogisticRegression(max_iter=100, random_state=0).fit(X, y)
print("The decision boundary of the multinomial logistic regression")
plot_decision_boundaries(X, y, clf)
The decision boundary of the multinomial logistic regression
Now, let us create a simple neural network with no hidden layers and train it for a few epochs.
At the end of each epoch, we will plot the decision boundaries using the BoundariesCallback defined below.
import tensorflow as tf
class BoundariesCallback(tf.keras.callbacks.Callback):
def __init__(self, X, y, clf, plot_freq=2):
self._X = X
self._y = y
self._clf = clf
self._plot_freq = plot_freq
def on_epoch_end(self, epoch, logs=None):
if epoch % self._plot_freq == 0:
plot_decision_boundaries(self._X, self._y, self._clf, step=epoch)
Below is the deep model that we will use in the rest of this section.
from tensorflow import keras
class DeepModel:
def __init__(self, n_units=[], activation_functions=[], learning_rate=0.005):
self._history = None
# This sequential model can be used to sequentially add layers.
self._model = keras.Sequential()
input_dim=2
# We set the linear layers according to the given parameters
for units, act in zip(n_units, activation_functions):
self._model.add(keras.layers.Dense(units=units, activation=act, input_dim=input_dim))
if input_dim is not None:
input_dim = None
self._model.add(keras.layers.Dense(units=1, input_dim=input_dim, activation='sigmoid'))
# This will output the final architecture of the model
self._model.summary()
# We compile the model with a specific loss and optimiser, it is here that the learning rate is set
self._model.compile(optimizer=tf.optimizers.Adam(learning_rate=learning_rate), loss='binary_crossentropy', metrics=['accuracy'])
def predict(self, X):
return np.round(self._model.predict(X))
def fit(self, X, y, callbacks=[], epochs=11, validation_split=0.2):
# If you don't want to see the training log, you can add verbose=0 to the fit method's arguments below.
self._history = self._model.fit(X, y, validation_split=validation_split, epochs=epochs,
batch_size=50, callbacks=callbacks)
return self._history
Now, we have the definition of our deep learning model, and we can compute its decision boundaries during training.
Below, we are going to create and train a very simple neural network with no hidden layers.
tf.random.set_seed(0)
no_hidden_layer = DeepModel()
# You can change how often the decision boundaries are displayed by modifying plot_freq below.
# Here we display the boundaries every 2 epochs, set it to 1 to display after every epochs for example.
callbacks = [BoundariesCallback(X, y, no_hidden_layer, plot_freq=2)]
# If you want to see how the model evolves after a more training time, you can increase the epochs parameter below
res = no_hidden_layer.fit(X, y, callbacks=callbacks, epochs=11)
Model: "sequential"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
dense (Dense) (None, 1) 3
=================================================================
Total params: 3
Trainable params: 3
Non-trainable params: 0
_________________________________________________________________
Epoch 1/11
123/160 [======================>.......] - ETA: 0s - loss: 0.5983 - accuracy: 0.7106
160/160 [==============================] - 1s 6ms/step - loss: 0.5692 - accuracy: 0.7387 - val_loss: 0.4473 - val_accuracy: 0.8595 Epoch 2/11 160/160 [==============================] - 0s 875us/step - loss: 0.3951 - accuracy: 0.8771 - val_loss: 0.3421 - val_accuracy: 0.8995 Epoch 3/11 81/160 [==============>...............] - ETA: 0s - loss: 0.3371 - accuracy: 0.8807
160/160 [==============================] - 1s 5ms/step - loss: 0.3286 - accuracy: 0.8788 - val_loss: 0.2957 - val_accuracy: 0.8940 Epoch 4/11 160/160 [==============================] - 0s 887us/step - loss: 0.2982 - accuracy: 0.8739 - val_loss: 0.2722 - val_accuracy: 0.8920 Epoch 5/11 151/160 [===========================>..] - ETA: 0s - loss: 0.2817 - accuracy: 0.8732
160/160 [==============================] - 1s 6ms/step - loss: 0.2819 - accuracy: 0.8727 - val_loss: 0.2583 - val_accuracy: 0.8915 Epoch 6/11 160/160 [==============================] - 0s 863us/step - loss: 0.2718 - accuracy: 0.8740 - val_loss: 0.2492 - val_accuracy: 0.8930 Epoch 7/11 152/160 [===========================>..] - ETA: 0s - loss: 0.2648 - accuracy: 0.8755
160/160 [==============================] - 1s 5ms/step - loss: 0.2649 - accuracy: 0.8754 - val_loss: 0.2426 - val_accuracy: 0.8930 Epoch 8/11 160/160 [==============================] - 0s 874us/step - loss: 0.2597 - accuracy: 0.8769 - val_loss: 0.2376 - val_accuracy: 0.8945 Epoch 9/11 156/160 [============================>.] - ETA: 0s - loss: 0.2543 - accuracy: 0.8809
160/160 [==============================] - 1s 6ms/step - loss: 0.2558 - accuracy: 0.8792 - val_loss: 0.2339 - val_accuracy: 0.8955 Epoch 10/11 160/160 [==============================] - 0s 881us/step - loss: 0.2529 - accuracy: 0.8805 - val_loss: 0.2309 - val_accuracy: 0.8970 Epoch 11/11 130/160 [=======================>......] - ETA: 0s - loss: 0.2528 - accuracy: 0.8800
160/160 [==============================] - 1s 6ms/step - loss: 0.2506 - accuracy: 0.8820 - val_loss: 0.2286 - val_accuracy: 0.8975
Let us repeat the last experiment with a hidden layer added to our neural network.
tf.random.set_seed(0)
# We add one hidden layer with 50 neurons and a linear activation.
# The linear activation is used by default.
one_hidden_layer = DeepModel(n_units=[50], activation_functions=[None])
# You can change how often the decision boundaries are displayed by modifying plot_freq below.
# Here we display the boundaries every 2 epochs, set it to 1 to display after every epochs for example.
callbacks = [BoundariesCallback(X, y, one_hidden_layer, plot_freq=2)]
# If you want to see how the model evolves after a more training time, you can increase the epochs parameter below
res = one_hidden_layer.fit(X, y, callbacks=callbacks, epochs=11)
Model: "sequential_1"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
dense_1 (Dense) (None, 50) 150
dense_2 (Dense) (None, 1) 51
=================================================================
Total params: 201
Trainable params: 201
Non-trainable params: 0
_________________________________________________________________
Epoch 1/11
108/160 [===================>..........] - ETA: 0s - loss: 0.3450 - accuracy: 0.8169
160/160 [==============================] - 1s 8ms/step - loss: 0.3111 - accuracy: 0.8400 - val_loss: 0.2219 - val_accuracy: 0.9025 Epoch 2/11 160/160 [==============================] - 0s 1ms/step - loss: 0.2452 - accuracy: 0.8836 - val_loss: 0.2224 - val_accuracy: 0.9035 Epoch 3/11 128/160 [=======================>......] - ETA: 0s - loss: 0.2433 - accuracy: 0.8863
160/160 [==============================] - 1s 6ms/step - loss: 0.2446 - accuracy: 0.8846 - val_loss: 0.2199 - val_accuracy: 0.9030 Epoch 4/11 160/160 [==============================] - 0s 988us/step - loss: 0.2444 - accuracy: 0.8840 - val_loss: 0.2211 - val_accuracy: 0.9000 Epoch 5/11 125/160 [======================>.......] - ETA: 0s - loss: 0.2445 - accuracy: 0.8842
160/160 [==============================] - 1s 8ms/step - loss: 0.2449 - accuracy: 0.8839 - val_loss: 0.2218 - val_accuracy: 0.9035 Epoch 6/11 160/160 [==============================] - 0s 1ms/step - loss: 0.2456 - accuracy: 0.8826 - val_loss: 0.2217 - val_accuracy: 0.9045 Epoch 7/11 118/160 [=====================>........] - ETA: 0s - loss: 0.2476 - accuracy: 0.8827
160/160 [==============================] - 1s 7ms/step - loss: 0.2441 - accuracy: 0.8836 - val_loss: 0.2196 - val_accuracy: 0.9035 Epoch 8/11 160/160 [==============================] - 0s 1ms/step - loss: 0.2445 - accuracy: 0.8838 - val_loss: 0.2198 - val_accuracy: 0.9015 Epoch 9/11 110/160 [===================>..........] - ETA: 0s - loss: 0.2477 - accuracy: 0.8856
160/160 [==============================] - 1s 7ms/step - loss: 0.2446 - accuracy: 0.8855 - val_loss: 0.2214 - val_accuracy: 0.9040 Epoch 10/11 160/160 [==============================] - 0s 1ms/step - loss: 0.2449 - accuracy: 0.8842 - val_loss: 0.2203 - val_accuracy: 0.8990 Epoch 11/11 128/160 [=======================>......] - ETA: 0s - loss: 0.2486 - accuracy: 0.8803
160/160 [==============================] - 1s 6ms/step - loss: 0.2449 - accuracy: 0.8839 - val_loss: 0.2222 - val_accuracy: 0.9000
Using a deep neural network with one hidden layer, let us now investigate the impact of activation functions on our hidden layer and the final predictions.
Here, we use the "ReLU" function, which stands for Rectified Linear Unit. This function transforms the outputs of your hidden layer by keeping only the positive part, so that $f(x) = \max(0, x)$. In this formula, $x$ is the net input and $f(x)$ is the activation.
tf.random.set_seed(0)
# We add one hidden layer with 50 neurons and relu activation function
one_hidden_layer_relu = DeepModel(n_units=[50], activation_functions=["relu"])
# You can change how often the decision boundaries are displayed by modifying plot_freq below.
# Here we display the boundaries every 2 epochs, set it to 1 to display after every epochs for example.
callbacks = [BoundariesCallback(X, y, one_hidden_layer_relu, plot_freq=2)]
# If you wish to see how the model evolves after more training time,
# you can increase the epochs parameter below.
res = one_hidden_layer_relu.fit(X, y, callbacks=callbacks, epochs=11)
Model: "sequential_2"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
dense_3 (Dense) (None, 50) 150
dense_4 (Dense) (None, 1) 51
=================================================================
Total params: 201
Trainable params: 201
Non-trainable params: 0
_________________________________________________________________
Epoch 1/11
123/160 [======================>.......] - ETA: 0s - loss: 0.3358 - accuracy: 0.8389
160/160 [==============================] - 1s 7ms/step - loss: 0.3109 - accuracy: 0.8493 - val_loss: 0.1950 - val_accuracy: 0.9165 Epoch 2/11 160/160 [==============================] - 0s 1ms/step - loss: 0.1954 - accuracy: 0.9015 - val_loss: 0.1553 - val_accuracy: 0.9355 Epoch 3/11 121/160 [=====================>........] - ETA: 0s - loss: 0.1459 - accuracy: 0.9337
160/160 [==============================] - 1s 6ms/step - loss: 0.1392 - accuracy: 0.9377 - val_loss: 0.0965 - val_accuracy: 0.9670 Epoch 4/11 160/160 [==============================] - 0s 1ms/step - loss: 0.0814 - accuracy: 0.9769 - val_loss: 0.0514 - val_accuracy: 0.9870 Epoch 5/11 129/160 [=======================>......] - ETA: 0s - loss: 0.0442 - accuracy: 0.9950
160/160 [==============================] - 1s 6ms/step - loss: 0.0411 - accuracy: 0.9959 - val_loss: 0.0257 - val_accuracy: 0.9985 Epoch 6/11 160/160 [==============================] - 0s 1ms/step - loss: 0.0211 - accuracy: 0.9998 - val_loss: 0.0144 - val_accuracy: 0.9995 Epoch 7/11 117/160 [====================>.........] - ETA: 0s - loss: 0.0135 - accuracy: 1.0000
160/160 [==============================] - 1s 6ms/step - loss: 0.0123 - accuracy: 1.0000 - val_loss: 0.0091 - val_accuracy: 1.0000 Epoch 8/11 160/160 [==============================] - 0s 1ms/step - loss: 0.0082 - accuracy: 1.0000 - val_loss: 0.0062 - val_accuracy: 1.0000 Epoch 9/11 120/160 [=====================>........] - ETA: 0s - loss: 0.0062 - accuracy: 1.0000
160/160 [==============================] - 1s 7ms/step - loss: 0.0059 - accuracy: 1.0000 - val_loss: 0.0046 - val_accuracy: 1.0000 Epoch 10/11 160/160 [==============================] - 0s 1ms/step - loss: 0.0044 - accuracy: 1.0000 - val_loss: 0.0036 - val_accuracy: 1.0000 Epoch 11/11 107/160 [===================>..........] - ETA: 0s - loss: 0.0038 - accuracy: 1.0000
160/160 [==============================] - 1s 7ms/step - loss: 0.0035 - accuracy: 1.0000 - val_loss: 0.0028 - val_accuracy: 1.0000
The default learning rate is 0.005. Change its value and observe and then analyse the results. You are encouraged to run this test with both linear and ReLU activations in the hidden layer. See the comments in the code below.
tf.random.set_seed(0)
# Choose your learning rate here
lr = 0.005
one_hidden_layer_lr = DeepModel(learning_rate=lr)
callbacks = [BoundariesCallback(X, y, one_hidden_layer_lr, plot_freq=2)]
# This network has linear units in the hidden layer.
res = one_hidden_layer_lr.fit(X, y, callbacks=callbacks, epochs=11)
# Use the line below to test the network with ReLU in the hidden layer. Both networks were defined above.
# res = one_hidden_layer_relu.fit(X, y, callbacks=callbacks, epochs=11)
Model: "sequential_3"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
dense_5 (Dense) (None, 1) 3
=================================================================
Total params: 3
Trainable params: 3
Non-trainable params: 0
_________________________________________________________________
Epoch 1/11
157/160 [============================>.] - ETA: 0s - loss: 0.5716 - accuracy: 0.7364
160/160 [==============================] - 1s 8ms/step - loss: 0.5692 - accuracy: 0.7387 - val_loss: 0.4473 - val_accuracy: 0.8595 Epoch 2/11 160/160 [==============================] - 0s 1ms/step - loss: 0.3951 - accuracy: 0.8771 - val_loss: 0.3421 - val_accuracy: 0.8995 Epoch 3/11 134/160 [========================>.....] - ETA: 0s - loss: 0.3303 - accuracy: 0.8800
160/160 [==============================] - 1s 7ms/step - loss: 0.3286 - accuracy: 0.8788 - val_loss: 0.2957 - val_accuracy: 0.8940 Epoch 4/11 160/160 [==============================] - 0s 1ms/step - loss: 0.2982 - accuracy: 0.8739 - val_loss: 0.2722 - val_accuracy: 0.8920 Epoch 5/11 147/160 [==========================>...] - ETA: 0s - loss: 0.2820 - accuracy: 0.8733
160/160 [==============================] - 1s 7ms/step - loss: 0.2819 - accuracy: 0.8727 - val_loss: 0.2583 - val_accuracy: 0.8915 Epoch 6/11 160/160 [==============================] - 0s 944us/step - loss: 0.2718 - accuracy: 0.8740 - val_loss: 0.2492 - val_accuracy: 0.8930 Epoch 7/11 137/160 [========================>.....] - ETA: 0s - loss: 0.2655 - accuracy: 0.8755
160/160 [==============================] - 1s 6ms/step - loss: 0.2649 - accuracy: 0.8754 - val_loss: 0.2426 - val_accuracy: 0.8930 Epoch 8/11 160/160 [==============================] - 0s 1ms/step - loss: 0.2597 - accuracy: 0.8769 - val_loss: 0.2376 - val_accuracy: 0.8945 Epoch 9/11 111/160 [===================>..........] - ETA: 0s - loss: 0.2587 - accuracy: 0.8795
160/160 [==============================] - 1s 7ms/step - loss: 0.2558 - accuracy: 0.8792 - val_loss: 0.2339 - val_accuracy: 0.8955 Epoch 10/11 160/160 [==============================] - 0s 855us/step - loss: 0.2529 - accuracy: 0.8805 - val_loss: 0.2309 - val_accuracy: 0.8970 Epoch 11/11 148/160 [==========================>...] - ETA: 0s - loss: 0.2496 - accuracy: 0.8831
160/160 [==============================] - 1s 6ms/step - loss: 0.2506 - accuracy: 0.8820 - val_loss: 0.2286 - val_accuracy: 0.8975
Now that you have seen how a deep model could be implemented using tensorflow 2, you will create your own implementation of a deep learning algorithm! We ask you to code the backpropagation algorithm that was presented in our lectures. All the equations required for this implementation are in our lecture slides. You will need to transfer them to your Python code below.
We have previously used the sigmoid or ReLU activation function in the last layer of our deep model, and we will need it again for this question. If we use sigmod in this section, we can reuse the equations that we have in our lecture slides.
def sigmoid(x):
# TODO: Comment the line below and implement me!
raise NotImplementedError("Implement the sigmoid function before testing.")
return 0
Now that our sigmoid function is ready, let us define a skeleton for our custom deep learning algorithm.
The backpropagation algorithm for a feedforward network of two layers of sigmoid units can be defined as follows:
For each $(x, y)$ in the training examples, DO:
This procedure is based on Table 4.2 in Machine learning, Tom Mitchell, McGraw-Hill Education, 1997. Note that the example in this book assumes that the output unit uses sigmoid activation and the SSE error is optimised. In our discussion of the delta rule with sigmoid activation, we assumed the CE error, which made the update equation of the output units slightly simpler. We used this assumption in the pseudocode above.
Using the class MyDeepModel below, do the following:
forward() method which corresponds to step 1 of the backpropagation algorithm.backward() method which corresponds to steps 2 and 3 of the backpropagation algorithm.update() method which corresponds to step 4 of the backpropagation algorithm.class Layer:
""" This is a convenience class that we will use to define layers in our deep learning model
"""
def __init__(self, input_dim, units, random_state):
self._random_state = random_state
# we initialise the weights randomly
self.weights = self._random_state.normal(size=(input_dim, units))
# we add the weight separately to retrieve it more easily afterwards.
self.bias = self._random_state.normal(size=units)
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score
class MyDeepModel:
# learning rate seems to be a sensitive parameter
def __init__(self, learning_rate=0.02):
self._random_state = np.random.RandomState(0)
self._history = []
self.learning_rate = learning_rate
self.layers = [
# This is the layer corresponding to something like
# self._model.add(keras.layers.Dense(10, input_dim=2, activatoin="sigmoid"))
# in our DeepModel
Layer(2, 10, self._random_state),
# This is the output layer corresponding to
# self._model.add(keras.layers.Dense(units=1, activation="sigmoid"))
# in our DeepModel. Note that now we pass the object sigmoid as parameter, not the string.
Layer(10, 1, self._random_state)
]
def predict(self, X):
""" Takes input data of shape (n_items, n_features) and returns an array of shape (n_items,)
containing the labels predicted for each input x
"""
y_pred = []
# Here we register the predictions of the model for each input
# and store the results in a list
for x in X:
y_pred.append(self.forward(x)[-1])
y_pred = np.round(np.array(y_pred))
return y_pred
def fit(self, X, y, epochs=11, validation_split=0.2):
""" Train the model during a given number of epochs and plot the decision boundaries after each epoch.
This method performs one step of the backpropagation algorithm for each data example
by calling the forward, backward and update methods.
"""
self._history = {"train": [], "test": []}
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=validation_split,
random_state=self._random_state)
# One epoch corresponds to the training steps needed to train the model on the whole dataset once
for i in range(epochs):
for xt, yt in zip(X_train, y_train):
# The forward step predicts the class of the input
try:
outputs = self.forward(xt)
# The backward step backpropagate the error
gradients = self.backward(outputs, yt)
# Now, we update the weights
self.update(xt, outputs, gradients)
except NotImplementedError as e:
print(e)
return
y_pred = self.predict(X_train)
acc_train = accuracy_score(y_train, y_pred)
self._history["train"].append(acc_train)
y_pred = self.predict(X_test)
acc_test = accuracy_score(y_test, y_pred)
self._history["test"].append(acc_test)
print("Accuracy at epoch {}:\ntrain: {}\ntest: {}".format(i, acc_train, acc_test))
plot_decision_boundaries(X, y, self, step=i)
return self._history
def forward(self, x):
""" Forward pass of the backpropagation
Takes the one data example as input and returns the output values of each layer
"""
outputs = []
# TODO: Comment the line below and implement me!
raise NotImplementedError("Implement the forward function before testing.")
return outputs
def backward(self, outputs, y_true):
""" Backward pass of the backpropagation algorithm
Takes the output of the layer obtained in forward pass, compute and backpropagate the error,
and returns the obtained gradients
"""
gradients = []
# TODO: Comment the line below and implement me!
raise NotImplementedError("Implement the backward function before testing.")
return gradients
def update(self, x, outputs, gradients):
""" Update the weights after the backward pass of the backpropagation algorithm
Takes the gradient obtained during the backpropagation and update the weights of each layer
"""
# TODO: Comment the line below and implement me!
raise NotImplementedError("Implement the update function before testing.")
return None
my_model = MyDeepModel()
print("Training my model")
res = my_model.fit(X, y, epochs=11)
Training my model Implement the forward function before testing.
tf_model = DeepModel(n_units=[10], activation_functions=["sigmoid"])
print("Training Tensorflow model")
callbacks = [BoundariesCallback(X, y, tf_model, plot_freq=1)]
res = tf_model.fit(X, y, callbacks=callbacks, epochs=11)
Model: "sequential_4"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
dense_6 (Dense) (None, 10) 30
dense_7 (Dense) (None, 1) 11
=================================================================
Total params: 41
Trainable params: 41
Non-trainable params: 0
_________________________________________________________________
Training Tensorflow model
Epoch 1/11
132/160 [=======================>......] - ETA: 0s - loss: 0.5290 - accuracy: 0.8314
160/160 [==============================] - 1s 7ms/step - loss: 0.5037 - accuracy: 0.8317 - val_loss: 0.3483 - val_accuracy: 0.8685 Epoch 2/11 137/160 [========================>.....] - ETA: 0s - loss: 0.3166 - accuracy: 0.8575
160/160 [==============================] - 1s 7ms/step - loss: 0.3102 - accuracy: 0.8599 - val_loss: 0.2573 - val_accuracy: 0.8885 Epoch 3/11 139/160 [=========================>....] - ETA: 0s - loss: 0.2651 - accuracy: 0.8786
160/160 [==============================] - 1s 5ms/step - loss: 0.2643 - accuracy: 0.8783 - val_loss: 0.2340 - val_accuracy: 0.8975 Epoch 4/11 145/160 [==========================>...] - ETA: 0s - loss: 0.2570 - accuracy: 0.8829
160/160 [==============================] - 1s 6ms/step - loss: 0.2527 - accuracy: 0.8856 - val_loss: 0.2274 - val_accuracy: 0.9015 Epoch 5/11 139/160 [=========================>....] - ETA: 0s - loss: 0.2518 - accuracy: 0.8846
160/160 [==============================] - 1s 6ms/step - loss: 0.2501 - accuracy: 0.8851 - val_loss: 0.2263 - val_accuracy: 0.9035 Epoch 6/11 140/160 [=========================>....] - ETA: 0s - loss: 0.2494 - accuracy: 0.8827
160/160 [==============================] - 1s 6ms/step - loss: 0.2496 - accuracy: 0.8834 - val_loss: 0.2259 - val_accuracy: 0.9040 Epoch 7/11 137/160 [========================>.....] - ETA: 0s - loss: 0.2505 - accuracy: 0.8842
160/160 [==============================] - 1s 6ms/step - loss: 0.2491 - accuracy: 0.8844 - val_loss: 0.2248 - val_accuracy: 0.9025 Epoch 8/11 142/160 [=========================>....] - ETA: 0s - loss: 0.2514 - accuracy: 0.8814
160/160 [==============================] - 1s 6ms/step - loss: 0.2488 - accuracy: 0.8834 - val_loss: 0.2246 - val_accuracy: 0.9025 Epoch 9/11 139/160 [=========================>....] - ETA: 0s - loss: 0.2465 - accuracy: 0.8882
160/160 [==============================] - 1s 6ms/step - loss: 0.2484 - accuracy: 0.8863 - val_loss: 0.2258 - val_accuracy: 0.9040 Epoch 10/11 114/160 [====================>.........] - ETA: 0s - loss: 0.2500 - accuracy: 0.8837
160/160 [==============================] - 1s 6ms/step - loss: 0.2487 - accuracy: 0.8861 - val_loss: 0.2241 - val_accuracy: 0.9020 Epoch 11/11 133/160 [=======================>......] - ETA: 0s - loss: 0.2495 - accuracy: 0.8842
160/160 [==============================] - 1s 6ms/step - loss: 0.2486 - accuracy: 0.8854 - val_loss: 0.2243 - val_accuracy: 0.9025
If you have finished early, and you'd like something more challenging to do, you can add a few options to refine your model. For example:
import datetime
print("Last modified: ", datetime.datetime.now().strftime("%d/%m/%Y %H:%M:%S") + "\n")
Last modified: 14/11/2022 21:25:02